Functions api ✨ 🗃️ #7539

wvangeit · 2025-04-16T10:19:14Z

What do these changes do?

This introduces the Functions API.
Functions are objects that can be in the backend: projects, solvers, python code, etc ...
Users can create their own functions from the relevant objects.
They accept parameters and return results.
Each function run creates a function job. These function job can also act as a cache to prevent rerunning a function with the same parameters.

Related issue/s

#7506

How to test

There are tests in the api-server (Rest) and web/server (RPC) service directories.

A high-level functional test script is here:
https://github.com/wvangeit/osparc-functions-api-test/blob/master/test.py

Dev-ops checklist

No ENV changes or I properly updated ENV (read the instruction)

codecov · 2025-04-16T10:20:54Z

Codecov Report

Attention: Patch coverage is 75.37500% with 197 lines in your changes missing coverage. Please review.

Project coverage is 86.03%. Comparing base (c552e46) to head (605fd28).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7539      +/-   ##
==========================================
- Coverage   87.57%   86.03%   -1.54%     
==========================================
  Files        1794     1731      -63     
  Lines       69368    68182    -1186     
  Branches     1136     1037      -99     
==========================================
- Hits        60747    58661    -2086     
- Misses       8311     9247     +936     
+ Partials      310      274      -36

Flag	Coverage Δ		*Carryforward flag
integrationtests	`59.88% <ø> (-4.85%)`	⬇️	Carriedforward from e032e1e
unittests	`86.64% <75.37%> (-0.25%)`	⬇️

*This pull request uses carry forward flags. Click here to find out more.

Components	Coverage Δ
api	`∅ <ø> (∅)`
pkg_aws_library	`∅ <ø> (∅)`
pkg_dask_task_models_library	`∅ <ø> (∅)`
pkg_models_library	`93.06% <100.00%> (+0.17%)`	⬆️
pkg_notifications_library	`85.26% <ø> (ø)`
pkg_postgres_database	`88.56% <100.00%> (+0.14%)`	⬆️
pkg_service_integration	`69.92% <ø> (ø)`
pkg_service_library	`72.02% <0.00%> (-0.87%)`	⬇️
pkg_settings_library	`∅ <ø> (∅)`
pkg_simcore_sdk	`65.71% <ø> (-19.96%)`	⬇️
agent	`96.46% <ø> (ø)`
api_server	`91.52% <78.69%> (-0.99%)`	⬇️
autoscaling	`96.08% <ø> (ø)`
catalog	`92.64% <ø> (ø)`
clusters_keeper	`99.25% <ø> (ø)`
dask_sidecar	`89.88% <ø> (ø)`
datcore_adapter	`98.12% <ø> (ø)`
director	`76.80% <ø> (ø)`
director_v2	`84.64% <ø> (-6.56%)`	⬇️
dynamic_scheduler	`96.76% <ø> (ø)`
dynamic_sidecar	`90.15% <ø> (ø)`
efs_guardian	`89.79% <ø> (ø)`
invitations	`93.28% <ø> (ø)`
payments	`92.63% <ø> (ø)`
resource_usage_tracker	`89.13% <ø> (+0.10%)`	⬆️
storage	`87.74% <ø> (ø)`
webclient	`∅ <ø> (∅)`
webserver	`85.01% <80.23%> (-0.86%)`	⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c552e46...605fd28. Read the comment docs.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bisgaard-itis

Thanks a lot for the great effort Werner! It looks very cool.
I suggest several minor changes, but I have one big request and that concerns running the function. I think it is crucial that that happens asyncronously and in one of our dedicated celery workers. This call (in the case of studies) was exactly what was blocking us last time we did the metamodeling. We are in a better position now because we have a job scheduling framework for handling "internal long running tasks". We should use it! 😁

packages/postgres-database/src/simcore_postgres_database/models/functions_models_db.py

...ages/service-library/src/servicelib/rabbitmq/rpc_interfaces/webserver/functions/functions.py

...ibrary/src/servicelib/rabbitmq/rpc_interfaces/webserver/functions/functions_rpc_interface.py

services/web/server/src/simcore_service_webserver/functions/_functions_controller_rpc.py

wvangeit · 2025-04-22T11:28:34Z

Thanks a lot for the great effort Werner! It looks very cool. I suggest several minor changes, but I have one big request and that concerns running the function. I think it is crucial that that happens asyncronously and in one of our dedicated celery workers. This call (in the case of studies) was exactly what was blocking us last time we did the metamodeling. We are in a better position now because we have a job scheduling framework for handling "internal long running tasks". We should use it! 😁

Yeah, sure makes sense. I'm basically calling the study_job api. It's probably best to implement the celery changes there? (it they're not there yet)

bisgaard-itis

Thanks a lot for the changes. I had another look and provided some more feedback. I think it is quite critical that we sort out the run function so that creating it becomes a long running task in a celery worker somewhere.

packages/models-library/src/models_library/api_schemas_api_server/functions.py

packages/models-library/src/models_library/api_schemas_webserver/functions_wb_schema.py

services/api-server/src/simcore_service_api_server/models/schemas/functions_api_schema.py

services/web/server/src/simcore_service_webserver/functions/_functions_repository.py

Copilot

Pull Request Overview

This PR introduces the Functions API to support registering, executing, and managing functions (across projects, solvers, and Python code) as well as caching their execution results. Key changes include the removal of the legacy webserver RPC controller, the addition of new API endpoints and service methods for functions handling, and the introduction of new schema models and database migrations for functions, function jobs, and their collections.

Reviewed Changes

Copilot reviewed 28 out of 30 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
services/web/server/src/simcore_service_webserver/functions/_controller_rpc.py	Removed the legacy RPC controller for functions since the functionality has been migrated.
services/api-server/tests/unit/api_functions/conftest.py	Added new test fixtures and mocks to support the Functions API.
services/api-server/src/simcore_service_api_server/services_rpc/wb_api_server.py	Extended the API client with methods for registering, getting, and managing functions and function jobs.
services/api-server/src/simcore_service_api_server/models/schemas/functions_api_schema.py	Introduced new schema models for the Functions API.
services/api-server/src/simcore_service_api_server/api/routes/studies_jobs.py	Refactored study job routes for clearer control flow and direct returns.
services/api-server/src/simcore_service_api_server/api/routes/solvers_jobs.py	Minor update with a noqa flag for improved linter compliance.
services/api-server/src/simcore_service_api_server/api/root.py	Updated the API router to include new endpoints with dedicated prefixes for functions, function jobs, and function job collections.
packages/service-library/src/servicelib/rabbitmq/rpc_interfaces/webserver/functions/functions_rpc_interface.py	Added a new RPC interface for functions, exposing methods for all functions-related operations.
packages/postgres-database/...	Added new database models and migrations to support functions, function jobs, and their collections.
packages/models-library/src/models_library/api_schemas_webserver/functions_wb_schema.py	Added new models and type aliases for the functions API exposed via the webserver.

Files not reviewed (2)

services/api-server/requirements/_test.in: Language not supported
services/api-server/requirements/_test.txt: Language not supported

Comments suppressed due to low confidence (1)

services/api-server/tests/unit/api_functions/conftest.py:299

[nitpick] Consider simplifying or aliasing the long module path used in the patch calls to improve readability and maintainability.

mock_wb_api_server_rpc.patch("servicelib.rabbitmq.rpc_interfaces.webserver.functions.functions_rpc_interface.register_function", backend_function_register.register_function)

packages/postgres-database/src/simcore_postgres_database/models/funcapi_function_jobs_table.py

pcrespov

OK. Let's get thisone in . but first, please check my suggestions and consider some changes.
If you have any questions, do not hesitate to drop be a line so we accelerate this PR merging.

TIP: activate mergify bot (write @mergify queue) so that once all tests pass, it makes sure that it merges.

...gres-database/src/simcore_postgres_database/models/funcapi_function_job_collections_table.py

.../simcore_postgres_database/models/funcapi_function_job_collections_to_function_jobs_table.py

services/api-server/src/simcore_service_api_server/api/routes/functions_routes.py

services/api-server/tests/unit/api_functions/conftest.py

services/api-server/tests/unit/api_functions/test_api_routers_functions.py

GitHK

I have a few more comments and suggestions.
Also a way to simplify the api-server's tests which will remove quite some code and make them more intuitive when reading.

packages/models-library/src/models_library/api_schemas_webserver/functions_wb_schema.py

...core_postgres_database/migration/versions/32a9ab5fa107_add_creation_modification_time_to_.py

...ibrary/src/servicelib/rabbitmq/rpc_interfaces/webserver/functions/functions_rpc_interface.py

services/web/server/tests/unit/with_dbs/04/functions_rpc/test_functions_controller_rpc.py

services/api-server/tests/unit/api_functions/conftest.py

services/api-server/tests/unit/api_functions/test_api_routers_functions.py

Copilot

Pull Request Overview

Enables the new Functions API end-to-end by removing legacy stubs, adding RPC client wrappers, Pydantic schemas, HTTP routes, database tables, and migrations.

Removes old webserver RPC controller for functions.
Adds RPC client methods in WbApiRpcClient and Pydantic schemas for functions, jobs, and collections.
Registers new /functions, /function_jobs, and /function_job_collections routes and creates SQL tables with migrations.

Reviewed Changes

Copilot reviewed 28 out of 30 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
services/web/server/src/simcore_service_webserver/functions/_controller_rpc.py	Removed old RPC router and ping endpoint
services/api-server/tests/unit/api_functions/conftest.py	Added fixtures and dummy RPC client for functions API tests
services/api-server/src/simcore_service_api_server/services_rpc/wb_api_server.py	Added wrapper methods for functions RPC in `WbApiRpcClient`
services/api-server/src/simcore_service_api_server/models/schemas/functions_api_schema.py	New Pydantic schemas for functions, jobs, and collections
services/api-server/src/simcore_service_api_server/api/root.py	Registered new routers for functions, jobs, and collections
packages/service-library/src/servicelib/rabbitmq/rpc_interfaces/webserver/functions/	RPC interface implementations for all functions endpoints
packages/postgres-database/src/simcore_postgres_database/models/funcapi_*.py	Added tables for functions, jobs, collections, and join
packages/postgres-database/src/simcore_postgres_database/migration/versions/*.py	Alembic scripts to create and evolve function API tables

Files not reviewed (2)

services/api-server/requirements/_test.in: Language not supported
services/api-server/requirements/_test.txt: Language not supported

Comments suppressed due to low confidence (2)

services/api-server/src/simcore_service_api_server/models/schemas/functions_api_schema.py:74

[nitpick] The FunctionJobCollection model uses id for its primary key field while other models use uid; consider renaming id to uid for consistency across your schemas.

class FunctionJobCollection(BaseModel):

services/api-server/src/simcore_service_api_server/services_rpc/wb_api_server.py:3

The imports dataclass and partial are unused in this file; please remove them to clean up dependencies.

from dataclasses import dataclass

.../simcore_postgres_database/models/funcapi_function_job_collections_to_function_jobs_table.py

matusdrobuliak66

Sorry didn't have time to go over it, but I do not want to block you so approve, I will go over it as soon as possible.

sanderegg

thanks, I added a few comments.
For next time please create smaller PRs that are easier to review (you will also get faster reviews):

1 PR with database changes
1 PR with API changes,
...

Thank you and good luck!

packages/models-library/src/models_library/api_schemas_webserver/functions_wb_schema.py

packages/postgres-database/src/simcore_postgres_database/models/funcapi_function_jobs_table.py

packages/postgres-database/src/simcore_postgres_database/models/funcapi_functions_table.py

pcrespov

Already approved.

GitHK

Thanks for all the changes!
I'm OK with it, just some minor things still remain.

...ibrary/src/servicelib/rabbitmq/rpc_interfaces/webserver/functions/functions_rpc_interface.py

services/api-server/src/simcore_service_api_server/models/schemas/functions_api_schema.py

services/web/server/tests/unit/with_dbs/04/functions_rpc/test_functions_controller_rpc.py

sonarqubecloud · 2025-05-15T09:06:23Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

wvangeit · 2025-05-15T09:14:31Z

@mergify queue

mergify · 2025-05-15T09:14:46Z

queue

✅ The pull request has been merged automatically

The pull request has been merged automatically at 67c3d67

wvangeit added 2 commits April 16, 2025 12:07

Add functions api. New commit to clean up db migration

1b1e8fe

Add db migration script for functions api

080e4fb

wvangeit added a:api framework api, data schemas, Feedback labels Apr 16, 2025

wvangeit added this to the Pauwel Kwak milestone Apr 16, 2025

wvangeit requested a review from bisgaard-itis April 16, 2025 10:19

wvangeit self-assigned this Apr 16, 2025

bisgaard-itis reviewed Apr 22, 2025

View reviewed changes

bisgaard-itis reviewed Apr 30, 2025

View reviewed changes

wvangeit added 19 commits May 6, 2025 11:13

Add functions api. New commit to clean up db migration

db68658

Add db migration script for functions api

0170982

Add solver functions

acdcfd4

Add default inputs to functions

8fbd814

Add default inputs to functions

b5cf30a

Add function collections

4e1ac9a

Working project function job collection

9fabebf

Add db migration for job collections

3c9f819

Adapt for changes in solver job api

77e0a00

Db merge heads functions draft and master rebase

b8722cc

Add tests for functions api server

efdd470

Add function rpc tests

be9ac7c

Move function rpc test dir

6a7ba23

Remove Nullable fields

23b36e9

Merge alembic heads after rebase

f956360

Fix tests after rebase

94c44c8

Add pagenation to function listing

61c3bbf

Fix function routes

9f3d968

Fix function pagination api tests

8619a2c

Copilot AI reviewed May 14, 2025

View reviewed changes

packages/postgres-database/src/simcore_postgres_database/models/funcapi_function_jobs_table.py Outdated Show resolved Hide resolved

pcrespov approved these changes May 14, 2025

View reviewed changes

pcrespov requested a review from Copilot May 14, 2025 12:50

GitHK reviewed May 14, 2025

View reviewed changes

Copilot AI reviewed May 14, 2025

View reviewed changes

.../simcore_postgres_database/models/funcapi_function_job_collections_to_function_jobs_table.py Show resolved Hide resolved

wvangeit added 3 commits May 14, 2025 18:48

Refactor function tests based on pr comments

1942489

Merge branch 'master' into functions_draft

b61d3c8

Split function route files and solve other comments

68d8fe5

matusdrobuliak66 approved these changes May 14, 2025

View reviewed changes

sanderegg approved these changes May 15, 2025

View reviewed changes

wvangeit added 3 commits May 15, 2025 09:29

Pylint fixes

5916524

Temp delete func db migration files

7e4d917

Add function db migration script

834395f

wvangeit requested review from GitHK, giancarloromeo, matusdrobuliak66, sanderegg and pcrespov May 15, 2025 07:55

pcrespov approved these changes May 15, 2025

View reviewed changes

sanderegg approved these changes May 15, 2025

View reviewed changes

wvangeit added 3 commits May 15, 2025 10:40

Fix comments from functions PR

ad08a53

Rm functions db migration to create a new one

267c7a5

Add a new function db migration script

e032e1e

GitHK approved these changes May 15, 2025

View reviewed changes

Fix latest functions PR comments

605fd28

wvangeit added the 🤖-automerge marks PR as ready to be merged for Mergify label May 15, 2025

mergify bot merged commit 67c3d67 into ITISFoundation:master May 15, 2025
92 of 96 checks passed

Functions api ✨ 🗃️ #7539

Functions api ✨ 🗃️ #7539

Uh oh!

Conversation

wvangeit commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Related issue/s

How to test

Dev-ops checklist

Uh oh!

codecov bot commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bisgaard-itis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wvangeit commented Apr 22, 2025

Uh oh!

bisgaard-itis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

pcrespov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GitHK left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

matusdrobuliak66 left a comment

Choose a reason for hiding this comment

Uh oh!

sanderegg left a comment

Choose a reason for hiding this comment

wvangeit commented Apr 16, 2025 •

edited

Loading

codecov bot commented Apr 16, 2025 •

edited

Loading

mergify bot commented May 15, 2025 •

edited

Loading